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Foreword 


For  rhany  years,  the  U.S.  Army  Materiel  Command  (AMC)  acquisition 
community  has  recognized  thaTthe  quality  of  a  contractor’s  performance  on  previ¬ 
ous  contracts  should' be  an  evaluation  factor  in  most  of  our  competitively  negoti¬ 
ated  acquisitions.  To  fully  realize  the  advantages  of  evaluating  past  performance, 
it  is  essential  that  we  consider  not  only  performance  information  that  offerors  may 
include  in  their  proposals,  but  also  information  obtained  from  other  sources.  Uni¬ 
form  AMC  procedures  for  obtaining  and  evaluating  past  performance  information 
have  been  developed  and  successfully  implemented. 

This  pamphlet  is  designed  to  help  you  participate  in  the  evaluation  of  past 
performance  during-.the  source  selection  process.  It  updates  the  guidance  devel¬ 
oped  during  an  ^initial  pilot  program,  and  contains  the  most  up-to-date  procedures 
for  policy  implementation.  These  procedures  are  based  in  large  part  on  the 
“lessons  learned"  and  suggestions  submitted  by  personnel  from  AMC  contracting 
activities  who  have  been  directly  responsible  for  the  successful  implementation  of 
the  program.  This  pamphlet  is  an  evolutionary  document  that  will  change  and 
improve  with  your  practical  suggestions  and  the  latest  revisions  to  Department  of 
Defense  and  Army  acquisition  policy. 


Keep  in  mind  that  policy  and  procedures,  no  matter  how  welLdevised,  are 
no  substitute  for  innovative  thinking  and  good  judgment.  This  pamphlet  provides 
a  basic  blueprint  for  conducting  past  performance  evaluations  within  the  tradi¬ 
tional  source  selection  process.  You  should  use  it,  not  rigidly,  but  as  basic  guid¬ 
ance  to  help  you  evaluate  past  performance  and  award  contracts  to  those  contrac¬ 
tors  who  will  deliver  quality  products  and  services,  on  time,  and  at  reasonable 
prices; 

I  extend  my  personal  thanks  to  the  members  of  the  Performance  Risk 
Assessment  Group  (PRAG)  Committee  who  were  responsible, for  developing  and 
implementing  the  PRAG  program.  I  also  extend  thanks  to  the  Major  Subordinate 
Command  (MSC)  representatives  who  actively  participated  in  the  PRAG  Work¬ 
shop  and  made  invaluable  contributions  to  the  drafting  of  this  pamphlet. 


firnmy  Dl  Ross 
Cfilmmaijding  General 
;rmy  Materiel  Command 


" Contractor  past  performance,  properly  documented  and 
considered,  is  a  powerful  tool  for  improving  the  quality 
and  timeliness  of  Defense  contracts. " 

—  Report  of  Joint  OSD-DOD  Component  Contractor 
Performance  Review  System  Task  Force 
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Program  Overview 


What  Is  Unique  About  This  Program? 


his  pamphlet  describes  an  innovative  way 
to  evaluate  contractor  past  performance 
during  the  source  selection  process.  AMC 
contracting  activities  helped  to  develop, 
test  and  implement  this  approach  which 
results  in  smarter  procurement  decisions 
and  better  contracts. 

The  distinctive  feature  of  this  program 
is  that  it  uses  information  that  is  outside 
of  the  offerors'  proposals  to  evaluate  past 
performance.  No  longer  must  contrac.ing 
activities  rely  solely  upon  the  very  contrac¬ 
tors  being  evaluated  for  past  performance 
information.  Now  contracting  officers  can 
use  independent  sources  of  information  to 
determine  how  well  those  contractors 
performed  in  the  past.  A  thorough  evalua¬ 
tion  of  past  performance  identifies  the 


relative  performance  risks  associated  with 
competing  projiosals  and  thereby  serves  to 
ensure  that  awards  are  made  to  good 
performers  rather  than  to  just  good  pro¬ 
posal  writers. 

A  second  unique  aspect  of  this  program 
is  that  it  provides  for  an  independent 
group  of  evaluators,  called  the  Perfor¬ 
mance  Risk  Assessment  Group  (PRAG >.  to 
evaluate  past  performance  separately  from 
the  proposal  evaluators.  This  approach  is 
simple  and  flexible.  The  traditional  source 
selection  process  remains  unchanged 
except  that  now  the  Source  Selection 
Authority  has  additional  information  from 
the  PRAG  that  will  result  in  better  award 
decisions. 


When  Does  This  Pamphlet  Apply? 


The  procedures  outlined  in  this  pamphlet 
apply  to  competitively  negotiated,  best 
value  procurements,  in  which  the  selection 
of  a  source  is  based  on  factors  other  than 
price  alone,  including  best  value  competi¬ 
tions  based  only  on  price  and  past  perfor¬ 
mance.  In  fact,  contracting  activities  can 
use  this  process  in  a  variety  of  procure¬ 
ments  including  services,  supply,  and 
research  and  development.  The  proce¬ 
dures  would  not  be  appropriate  in  negoti¬ 
ated  procurements  when  the  award  is 


based  solely  on  low  price,  and  they  must 
not  be  used  to  circumvent  the  Certificate  of 
Competency  procedures  for  small  busi¬ 
nesses. 

While  the  Office  of  Federal  Procure¬ 
ment  Policy  (OFPP)  Policy  Letter  92-0 
dated  December  30.  1992.  requires  the 
evaluation  of  past  performance  on  all 
competitively  negotiated  acquisitions  o\er 
$100,000.  the  procedures  outlined  in  this 
pamphlet  are  optional  for  acquisitions 
under  $10,000,000. 


How  Does  This  Approach  Differ  From 
Pre-Award  Surveys? 


t  is  important  to  distinguish  past  perfor¬ 
mance  evaluations  from  pre-award 
surveys.  The  Defense  Logistics  Agency 
conducts  pre-award  surveys  to  determine 
whether  a  contractor  is  responsible. 
Responsibility  is  a  broad  concept  that 
addresses  whether  a  contractor  has  the 
capability  to  perform  a  particular  contract 
based  upon  an  analysis  of  many  areas 
including  financial  resources,  operational 
controls,  technical  skills,  quality  assur¬ 
ance  and  past  performance.  These  sur¬ 
veys  provide  a  "yes/no."  "pass/fail,"  or 
"go/no-go"  answer  to  the  question,  can 
this  offeror  do  the  work? 

Unlike  a  pre-award  survey,  a  past 


performance  evaluation  daring  the  source 
selection  process  is  a  very  specific  endeavor 
that  seeks  to  identify  the  degree  of  risk 
associated  with  each  competing  offeror, 
thereby  permitting  a  comparative  assess¬ 
ment  of  offers.  Rather  than  asking 
whether  an  offeror  can  do  the  work,  a  past 
performance  evaluation  asks,  will  it  do  that 
work  successfully?  In  short,  it  describes 
the  degree  of  confidence  the  government 
has  in  the  offeror's  likelihood  of  success.  If 
properly  conducted,  the  past  performance 
evaluation  and  the  pre-award  survey  will 
complement  each  other  and  provide  a  more 
complete  picture  of  an  offeror  then  either 
one  could  by  itself. 


- PRAG  Structure, - 

Composition  and  Evaluation 


What  Is  The  Function  Of  The  PRAG? 


The  PRAG  is  responsible  for  conducting  the 
past  performance  evaluation  to  determine 
the  degree  of  risk  involved  in  accepting  a 
contractor's  promises  of  performance. 

This  determination  is  called  performance 


risk.  The  PRAG  prepares  a  report  that 
describes  these  risk  assessments  and 
identifies  strong  and  weak  points  in  each 
offeror's  past  performance. 


3 


What  Is  The  Proper  PRAG  Composition  And 
Structure? 


Each  contracting  activity  determines  the 
appropriate  membership  and  structure  of 
its  PRAGs.  The  quality  of  the  PRAG 
report  depends  upon  the  quality  of  the 
PRAG.  Ideally,  the  membership  should 
include  individuals  who  have  procurement, 
cost,  and  technical  expertise  as  well  as 
PRAG  experience.  The  individuals  se¬ 
lected  should  also  be  capable  of  making 
sound  and  impartial  judgments. 

The  heart  of  the  PRAG  assessment  is 
the  information  gathering  process. 

Through  questionnaires  and  telephone 
interviews,  the  PRAG  can  obtain  a  detailed 
and  useful  picture  of  an  olferor's  past 
performance.  Because  of  the  importance  of 
the  information  gathering  process,  it  is 
absolutely  critical  that  PRAG  members 
have  the  ability  to  conduct  meaningful 
telephone  interviews.  They  should  also  be 
able  to  assimilate  voluminous  data,  exer¬ 
cise  sound  judgment,  arrive  at  conclusions 
that  make  common  sense,  and  communi 
cate  those  conclusions  effectively  both 
orally  and  in  writing. 

The  size  of  the  PRAG  should  reflect  the 


number  of  contractors  and  subcontractors 
expected  to  respond  to  the  solicitation  as 
well  as  the  nature  and  complexity  of  the 
solicitation  requirements.  Experience 
indicates  that  a  four-person  team,  includ¬ 
ing  one  administrative  assistant,  is  a 
reasonable  size  for  a  solicitation  with  three 
to  six  offerors.  The  best  practice  is  to  have 
at  least  two  members,  one  with  procure¬ 
ment  expertise  and  one  with  technical 
expertise,  on  each  PRAG  to  allow  for 
dialogue,  brainstorming,  and  in-depth  fact 
finding. 

The  PRAG  structure  should  enhance 
its  ability  to  independently  evaluate 
performance  risk.  The  PRAG  may  operate 
separately  from  the  Source  Selection 
Evaluation  Board  (SSEB)  and  report 
directly  to  the  Source  Selection  Advisory 
Council  (SSAC),  or  it  may  operate  as  a 
separate  SSEB  subgroup  that  reports  to 
the  SSEB  chairperson.  A  PRAG  assess¬ 
ment  plan,  like  the  sample  at  appendix  C, 
should  be  developed  early  in  the  process 
and  made  a  part  of  the  source  selection 
plan. 


What  Subfactors  Should  Be  Used? 


The  contracting  activity  has  wide  latitude  in 
selecting  evaluation  factors  and  subfactors. 
The  past  performance  subfactors,  if  any, 
need  not  mirror  those  of  the  proposal 
evaluation.  In  most  cases  the  PRAG  at 
least  considers  the  offeror's  record  for  on 
time  delivery,  good  technical  quality,  and 
cost  control  to  determine  its  likelihood  of 
success  in  performing  the  solicitation's 
requirements; 


OFPP  Policy  Letter  92-5  suggests  that 
past  performance  include  the  contractor  s 
record  of  conforming  to  specifications  and 
to  standards  of  good  workmanship;  the 
contractor's  record  of  containing  and 
forecasting  costs  on  any  previously  per¬ 
formed  cost  reimbursable  contracts;  the 
contractor's  adherence  to  contract  sched¬ 
ules,  including  the  administrative  aspects 
of  performance;  the  contractor's  history  for 


reasonable  and  coo|>erntive  behavior  and  when  considering  past  performance,  not  a 

commitment  to  customer  satisfaction,  and  separate  element  of  past  performance, 

generally,  the  contractor's  business-like  Irrelevant  past  performance  should  not 

concern  for  the  interests  of  the  cusiomcr.  form  the  basis  of  a  performance  risk  assess- 
Relevancy  should  not  be  described  as  a  ment. 
subfactor.  Relevancy  is  a  threshold  question 


How  Much  Weight  To  Give  Past  Performance 


Past  performance  should  be  given  sufficient 
evaluation  weight  to  ensure  that  it  is  mean¬ 
ingfully  considered  throughout  the  source 
selection  process  and  will  be  a  valid  discrimi¬ 
nator  among  the  offers  received. 

The  government  can  evaluate  both  the 
offeror's  experience  and  past  performance. 
.However,  it  is  improper  to  exaggerate  the 


importance  of  past  performance  by  first 
evaluating  it  separately  and  then  again  as 
an  aspect  of  experience.  Simply  put,  nei¬ 
ther  past  performance  nor  experience 
should  be  evaluated  twice.  It  is  proper, 
however,  to  distinguish  company  experi¬ 
ence  from  personnel  experience  and  evalu¬ 
ate  both. 


What  Are  The  Rating  Categories? 


The  PRAG  may  use  the  following  definitions  of  the  oxTeror's  performance  record,  little 
performance  risk  to  describe  the  results  of  its  doubt  exists  that  the  offeror  will  success- 
assessment:  fully  perform  the  required  effort. 

•  High  Performance  Risk  -  Based  on  •  Unknown  Performance  Risk  -  No 

the  offeror's  performance  record,  significant  performance  record  identifiable  (This 
doubt  exists  that  the  offeror  will  successfully  category  is  optional.  See  ‘‘How  to  Evaluate 
perform  the  required  effort.  No  Past  Performance,”  page  6). 

•  Moderate  Performance  Risk  -  Based 

on  the  offeror's  performance  record,  some  Note:  Each  of  the  high,  moderate,  and 

doubt  exists  that  -he  offeror  will  successfully  low  risk  categories  may  be  further  subdi- 

perform  the  required  effort.  vided  to  enhance  the  comparative  analysis 

•  Low  Performance  Risk  -  Based  on  of  offerors. 


How  To  Evaluate  No  Past  Performance 


In  most  cases  the  PRAG  will  find  some 
related  past  performance  information  for 
each  contractor  and  subcontractor,  espe¬ 
cially  if  the  PRAG  applies  a  broad  inter¬ 
pretation  of  relevancy.  Occasionally, 
however,  a  PRAG  cannot,  find  any  rel¬ 
evant  information.  In  those  cases,  con¬ 
tracting  activities  should  treat  an 
offeror’s  lack  of  past  performance  as  an 
unknown  performance  risk  that  is  neu¬ 
tral.  having  no  positive  or  negative  evalu¬ 
ative  significance.  This  approach  allows 
the  government  to  evaluate  past  perfor¬ 
mance  in  a  manner  that  is  fair  to  newcom¬ 
ers. 

An  alternative  approach  may  be  used 
on  rare  occasions  when  the  government 


must  have  a  contractor  with  a  proven 
performance  record.  In  this  situation,  an 
offeror  with  no  related  past  |>erformanco 
may  represent  a  high  or  moderate  perfor¬ 
mance  risk  to  the  contracting  activity.  This 
alternative  approach  should  only  be  used  if 
experience  is  evaluated  by  the  PRAG.  not 
the  SSEB.  In  this  case,  the  solicitation 
should  clearly  state  that  the  PRAG  will 
evaluate  experience  as  well  as  past  perfor- 
mancc  and  that  a  lack  of  experience  may 
result  in  a  high  or  moderate  risk  rating. 
Even  here  the  government  can  ease  the 
impact  on  newcomers  by  including  lan¬ 
guage  in  the  solicitation  that  encourages 
them  to  team  with  proven  performers. 


What  To  Include  In  The  Solicitation 


The  solicitation  should  clearly  state  that: 

(1)  the  government  will  conduct  a  perfor¬ 
mance  risk  assessment  based  upon  the 
past  performance  of  the  oiferors  and  their 
proposed  subcontractors  as  it  relates  to 
the  probability  of  successful  accomplish¬ 
ment  of  the  work  required  by  the  solicita¬ 
tion; 

(2)  in  conducting  the  performance  risk 
assessment,  the  government  may  use  data 
.provided  by  the  offeror  and  data  obtained 
from  other  sources; 

(3)  while  the  government  may  elect  to 
consider  data  obtained  from  other  sources, 
the  burden  of  providing  thorough  and 
complete  past  performance  information 
rests  with  the  offeror.  Appendix  A  con¬ 
tains  a  sample  Section  M  provision  for  use 
in  solicitations. 


Section  Lof  the  solicitation  should 
instruct  offerors  to  submit  information 
concerning  contracts  and  subcontracts 
which  are  in  any  way  similar  to  the  work 
required  by  the  solicitation,  or  which 
offerors  consider  relevant  in  demonstrat¬ 
ing  their  ability  to  perform  the  proposed 
effort.  Also,  it  is  important  that  the  offeror 
specifically  describe  the  work  that  its 
subcontractors  will  perform  so  that  the 
PRAG  can  conduct  a  meaningful  jx?rfor- 
mnnee  risk  assessment  on  each  significant 
subcontractor.  Appendix  B  contains  a 
sample  Section  L  provision  for  use  in 
solicitations. 

Presolicitation  or  preproposal  confer¬ 
ences  should  explain  the  jxrformance  risk 
methodology  to  ensure  that  offerors  under¬ 
stand  the  process  and  its  overall  signifi¬ 
cance. 


PRAG  Operations 


How  To  Begin 


The  PRAG  chairperson  should  hold  a  meeting 
of  PRAG  members  as  soon  as  possible  prior  to 
the  receipt  of  proposals  to  outline  the  PRAG 
process,  obtain  signed  nondisclosure  state¬ 
ments,  and  distribute  the  evaluation  plan 
and  Request  for  Proposal  (RFP).  The  requir¬ 
ing  activity  should  brief  the  PRAG  on  the 
technical  requirements  of  the  acquisition. 

The  PRAG  chair  may  assign  each  PRAG 
member  an  offeror(s)  for  whom  they  will 


screen  the  available  data  to  select  the  most 
recent  and  relevant  references  for  in-depth 
fact  finding.  However,  some  contracting 
activities  prefer  to  assign  the  work  by 
functional  area  rather  than  by  offeror.  In 
either  event,  the  PRAG  members  will  meet 
after  gathering  past  performance  informa¬ 
tion,  to  determine  the  performance  risk 
ratings. 


What  Sources  Of  Data  Are  Available? 


A  MC  established  a  centralized  networking 
system  to  provide  sources  of  past  perfor¬ 
mance  information  to  the  PRAG.  This  device 
is  called  the  Contractor  Information  System 
(CIS),  and  it  contains  only  factual  informa¬ 
tion  pertaining  to  contractors.  The  CIS  is 
basically  an  electronic  telephone  book.  It 
does  not  contain  report  cards  on  contractors. 
Questions  pertaining  to  the  CIS  or  the  FRAG 
methodology  that  cannot  be  answered  by  the 
source  selection  hierarchy  at  the  MSC  may 
be  referred  to  the  AMC  Deputy  Chief  of  Staff 
for  Acquisition. 

Although  the  PRAG  may  consider  data 
available  from  many  sources  such  as  DLA 
and  AMC,  its  main  sources  of  information  are 
often  the  references  cited  by  offerors  in  their 
proposals.  Upon  receipt  of  proposals  and  the 
AMC  report,  the  PRAG  will  determine  which 
of  the  offeror’s  past  contract  efforts  relate  to 
the  solicitation  requirements.  Although 
these  determinations  of  relevancy  are  judg¬ 
ment  calls,  it  is  helpful  to  consider  the 
offeror’s  explanation  of  relevancy  contained 
in  its  proposal. 


Ir.  some  cases,  previous  contracts  as  a 
whole  may  be  similar  to  the  current  con¬ 
tract  while  in  others  only  portions  of  previ¬ 
ous  contracts  may  be  relevant. 

For  example,  the  government  uses  Ada 
software  language  in  many  different  sys¬ 
tems.  If  a  solicitation  calls  for  the  develop¬ 
ment  of  Ada  software  for  an  aircraft  sys¬ 
tem,  the  contractor  might  identify  a  previ¬ 
ous  effort  where  it  developed  Ada  software 
for  a  satellite  terminal.  The  government 
may  consider  that  previous  effort  to  be 
relevant  for  purposes  of  assessing  the 
contractor’s  ability  to  develop  Ada  software 
even  though  the  underlying  system  is 
different  from  the  current  requirement. 
Another  example  is  the  evaluation  of  the 
contractor’s  management,  planning,  and 
scheduling  of  subcontractors  on  a  past 
service  contract  for  a  current  production 
requirement  calling  for  integration  skills. 

The  PRAG  should  consider  the  most 
recent  data  available.  The  best  practice  is 
to  select  efforts  that  are  either  still  ;n 
progress  or  just  completed,  and  that  have 


at  least  1  year  of  performance  history. 

The  actual  cut-off  time  is  left  blank  in  the 
sample  Section  L  provision  in  appendix  B 
because  it  should  be  determined  by  the 
contracting  officer  on  a  case-by-case  basis. 
However,  most  activities  have  used  3 
years  with  much  success. 

It  is  noted  that  OFPP  Policy  Letter  92- 
5  dated  December  30, 1992,  not  only 
requires  the  evaluation  of  past  perfor¬ 
mance  during  the  source  selection  process, 


but  also  requires  contracting  activities  to 
prepare  an  evaluation  of  contractor  perfor¬ 
mance  at  the  time  the  work  under  the 
contract  in  completed  for  each  contract  in 
excess  of  $100,000.  These,  latter  evalua¬ 
tions  arc  then  placed  in  the  contract  file. 
The  PRAGs  should  use  them  during  the 
source  selection  process  to  help  arrive  at 
their  own  assessment  of  an  offeror's  past 
performance. 


Can  The  PRAG  Use  Commercial  References? 


he  best  practice  is  to  rely  on  government 
sources  of  information.  However,  it  is 
permissible  to  use  nongovernment  refer¬ 
ences  when  necessary-  The  PRAG  should 
verify  information  received  from  commer¬ 


cial  and  foreign  government  sources  to 
ensure  accuracy.  The  use  of  such  refer¬ 
ences  for  one  offeror does  not  require  the 
same  for  all  offerors  so  long  as  sufficient 
information  is  available  for  them. 


How  To  Collect  Information 


he  PRAG  gathers  information  using 
questionnaires,  telephonic  inquiries,  or 
both.  Field  experience  indicates  that 
questionnaires  provide  useful  but  incom¬ 
plete  information.  Ahelpful  approach  is 
to  start  by  sending  a  common  question¬ 
naire  to  each  reference  and  to  conclude  by 
calling  those  who  respond  with  the  most 


promising  information.  Experience  indi¬ 
cates  that  whether  you  send  question¬ 
naires  or  not,  you  will  most  likely  conclude 
by  calling  the  reference  to  obtain  more 
detail  or  clarification. 

Samples  of  questions  for  telephone 
interviews  and  written  questions  are 
included  in  appendixes  F  and  G. 


Where  To  Conduct  Telephone  Interviews 


Following  the  screening  of  prev  .us  contracts 
for  further  in-depth  review,  each  PRAG 
member  should  send  questionnaires  and/or 
initiate  telephone  calls  to  the  identified 
references  for  those  efforts.  The  interview¬ 
ing  and  reporting  of  results  are. usually 
individual  efforts  conducted  by  each  PRAG 
member.  However,  it  is  sometimes  helpful 
for  the  PRAG  to  collect  information  as  a 
group  through  the  use  of  conference  calls. 

In  any  event,  the  environment  in  which  this 
work  is  done  significantly  impacts  both  the 
time  required  to  complete  tl:'  :.  portion  of  the 
process  and  the  quality  of  the  results.  These 
activities  are  hampered  severely  if  each 
PRAG  member  attempts  tp  conduct  tele¬ 


phone  interviews  at  their  normal  work  site 
with  all  of  its  attendant  interruptions, 
distractions,  and  security  risks. 

If,  on.  the  other  hand,  the  PRAG  mem¬ 
bers  are  able  to  assemble  as  a  group  for 
telephone  interviews,  they  will  be  able  to 
provide  considerable  reinforcement  and 
instant  feedback  for  one  another.  Each 
PRAG  member  should  be  able  to  devote 
their  undivided  attention  to  this  initial 
assessment  process.  Although  this  ap¬ 
proach  requires  a  secure  area  that  is  large 
enough  to  accommodate  all  of  the  PRAG 
members,  the  resulting  benefits  are  signifi¬ 
cant. 


How  To  Conduct  Telephone  Interviews 


T  he  telephone  interview  process  is  an  art  matches  that  of  the  reference. 

form.  Until  a  smooth  conversation  pattern  is  Prior  to  initiating  a*telephone  inter- 
developed,  it  is  an  inherently  uncomfortable  view,  a  PRAG  member  should  gather  all 
situation  for  many  people.  There  will  be  available  information  on  a  specific  effort 

some  difficulty  learning  how  to  start  a  tele-  and  draft  a  list  of  questions.  There  may  be 

phone  interview,  keep  it  moving,,  and  cover  a  common  group  of  questions  for  all  offerors 

all  important  areas.  As  the  interviewing  and/or  tailored  questions  for  each  offeror, 
process  continues,  the  PRAG  member  usu-  depending.upon  the  circumstances.  These 
ally  uncovers  special  items  of  interest  that  questions  can  either  be  sent  as  question- 

he  or  she  will  want  to  pursue  through  follow  naires  to  each  reference  or  be  used  by  the 

up  calls:  PRAG  member  during  the  telephone  inter- 

At  least  two  references  should  be  con-  view, 
tacted  oh  each  previous  contract  effort  se-  At  the  start  of  each  telephone  interview, 

lected  for  in-depth  review.  Additional  refer-  the  PRAG  member  should  explain  the 
ences  are  often- identified  during  the  inter-  purpose  of  the  call  and  request  voluntary 
views.  Maximum  effectiveness  occurs  when  assistance  from  the  reference.  The  PRAG 

the  expertise  of  the  PRAG  interviewer  member  should  explain  that  he  or  she  will 


document  the  results  of  the  conversation 
and  send  a  copy  of  the  memorandum  to 
the  reference  for  verification.  There  is 
usually  no  need  to  divulge  the  solicitation 
number,  program  description,  or  other 
identifying  information  to  the  reference. 

If  you  do  so,  you  need  to  obtain  a  nondis¬ 
closure  statement. 

In  most  instances  the  reference  will 
willingly  provide  the  information  re¬ 
quested.  In  those  rare  cases  when  the 
reference  refuses  to  participate,  the  PRAG 
member  should  request  assistance 
through  the  source  selection  hierarchy  at 
the  MSC.  Alternatively,  the  PRAG  mem¬ 
ber  may  attempt  to  continue  the  interview 


off-the-record  to  obtain  data  that  may  be 
validated  on-the-record  during  interviews 
with  other  references. 

It  is  important  to  pursue  the  underly¬ 
ing  facts  supporting  any  conclusionary 
statements  received  on  a  contractor, 
particularly  if  they  are  unusually  positive 
or  negative.  The  PRA.G  member  can. 
determine  neither  the  magnitude  of  a 
reported  problem  nor  its  possible  impact 
on  the  current  risk  assessment  without 
first  understanding  the  details  surround¬ 
ing  the  problem.  It  is  helpful  for  the 
PRAG  members  to  meet  periodically  to 
share  information  and  ideas. 


How  To  Document  Telephone  Interviews 


Immediately  following  a  telephone  inter¬ 
view,  the  PRAG  member  must  prepare  a 
narrative  summary  of  the  conversation 
and  send  it  to  the  reference  for  verifica¬ 
tion  preferably  by  certified  mail  return 
receipt  requested.  Datafax  transmissions 
are  also  acceptable.  The  following  step  is 
extremely  important.  Extra  care  must  be 
taken  to  ensure  accuracy,  clarity,  and 
legibility  because  these-summaries  often 
represent  the  only  written  back-up  sup¬ 
porting  the  opinions  and  conclusions  of 
the  final  PRAG  assessment  report. 

In  order  to  maintain  accurate  records 
and  facilitate  verification,  the  telephone 
record  form  should  include  the  reference's 
name,  full  mailing  address  and  telephone 
number,  the  date  and  time  of  the  call,  and 
the  description  of  the  contract  effort 
discussed.  A  sample  telephone  record 
form  is  attached  at  appendix  D. 

The  PRAG  should  send  the  telephone 


memorandum  to  the  reference,  stating 
explicitly  that  if  the  reference  does  not 
object  to  its  content  within  the  time  speci¬ 
fied,  it  will  be  accepted  as  correct.  The 
amount  of  time  allowed  for  a  response 
depends  on  the  circumstances  of  each 
procurement.  A  sample  cover  letter  is 
attached  at  appendix  E.  Note  that  the 
reference  need  not  sign  a  nondisclosure 
form  if  the  PRAG  member  withholds  the 
identity  of  the  program  and  solicitation 
number. 

If  a  reference  indicates  that  the  narra¬ 
tive  is  incorrect,  then  a  corrected -narrative 
must  be  sent  for  verification.  Experience 
ndicates  that  in  most  instances,  changes 
are  minor.  If,  however,  a  reference  ex¬ 
presses  opposition  to  a  record  and  satisfac¬ 
tory  corrections  cannot  be  agreed  ujxm,  the 
PRAG  should  not  rely  on  the  record. 
Another  source  may  provide  the  same 
information,  however. 


How  To  Assign  Performance  Risk  Ratings 


On^e  the  telephone  interviews  are  completed, 
thb  entire  PRAG  needs  to  assess  all  offerors 
and  assign  performance  risk  ratings.  The 
PRAG  should  note  instances  of  singularly 
good  or  poor  performance  and  relate  it  to  the 
solicitation  requirements.  Once  again,  it  is 
helpful  for  the  PRAG  to  review  the  statement 
of  work  and  specifications.  If  the  PRAG 
identifies  past  performance  problems  on  a 
prior  contract,  it  should  consider  the  role 
government  fault  played  in  that  result. 

The  PRAG  should  not  limit  its  inquiry 
solely  to  the  proposing  entity  if  other  corpo¬ 
rate  divisions,  contractors  or  subcontractors 
will  perform  a  critical  element  of  the  pro¬ 
posed  effort.  The,  performance  record  of  those 
organizations  should  be  assessed  in  accor¬ 
dance  with  the  solicitation.  Performance  risk 
assessments  should  consider  the  number  and 
severity  of  problems,  the  demonstrated 
effectiveness  of  corrective  actions  taken  (not 


just  planned  or  promised),  and  the  overall 
work  record. 

The  PRAG’s  assessment  is  usually 
based  upon  subjective  judgment.  It  is  not  a 
precise  or  mechanical  process.  The  assess¬ 
ment  should  include  a  description  of  the 
underlying  rationale  for  the  conclusions 
reached.  As  long  as  that  rationale  is 
reasonable,  it  will  withstand  scrutiny  even 
if  other  reasonable  conclusions  exist. 

A  word  of  caution  is  appropriate  con¬ 
cerning  offeror  promises  to  correct  past 
performance  failures,  as  opposed  to  actions 
already  taken  to  correct  such  failures.  A 
promise  to  improve  does  not,  by  itself, 
improve  past  performance.  However, 
demonstrated  corrective  actions  reflect  a 
commitment  to  rectify  past  performance 
problems,  and  therefore,  can  lower  the  risk 
of  similar  performance  failures. 


What  To  Include  In  The  Assessment  Report 


he  goal  is  to  avoid  saying  too  much  or  too 
little  in  the  PRAG  report.  Although  there  is 
no  need  to  restate  everything  contained  in 
the  telephone  memoranda,  the- PRAG  must 
provide  the  source  selection  authority  with 
that  information  needed  to  make  informed 
judgments.  Conclusionary  statements  must 
be  supported  by  the  underlying  factual  basis 
The  best  practice  is  to  state  the  conclusion 
and  provide  specific  examples  that  support 
that  conclusion. 

To  ensure  that  the  risk  assessments 


provide  the  necessary  background  informa¬ 
tion  and  are  structured  consistently,  the 
entire  PRAG  should  review  and  evaluate 
the  report  on  each  offeror.  During  this 
review,  the  PRAG  should  correct  state¬ 
ments  that  appear  unsupported,  inconsis¬ 
tent,  or  unnecessary. 

Occasionally  the  PRAG  will  be  unable 
to  arrive  at  a  unanimous  agreement  on  a 
particular  risk  assessment.  If  this  occurs, 
the  PRAG  may  include  the  dissenting 
opinion  as  part  of  the  assessment  report. 
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Should  The  PRAG  Brief  The  Results? 


The  PRAG’s  submission  of  the  assessment 
report  usually  completes  the  major  portion 
of  its  work.  However,  because  the  PRAG 
concept  is  relatively  new,  the  PRAG  chair 
should  remind  the  source  selection  offi¬ 
cials  of  the  importance  and  purpose  of  the 


PRAG  to  ensure  that  everyone  fully  com¬ 
prehends  the  significance  of  the  results 
being  briefed.  Experience  reveals  that 
source  selection  officials  are  more  apt  to 
rely  upon  PRAG  results  if  they  thoroughly 
understand  the  process. 


Should  PRAG  Findings  Be  Disclosed  To  Offerors? 


During  discussions  with  offerors  in  competi¬ 
tively  negotiated  procurements,  the  con¬ 
tracting  officer  must  disclose  deficiencies 
in  the  offerors’  proposals.  Arguably, 
negative  past  performance  information 
provided  by  a  reference  is  generally  not  a 
“proposal  deficiency"  because  it  is  based 
upon  information  outside  of  that  proposal. 
Nonetheless,  a  past  performance  problem 
can  be  a  significant  shortcoming  that 
must,  in  fairness,  be  brought  to  the  atten¬ 
tion  of  the  offeror  during  discussions  if  the 
offeror  has  not  previously  been  apprised  of 
the  problem  and  provided  an  opportunity 
to  respond. 

This  practice  validates  any  negative 
information  relied  upon  during  the  risk 
assessment  process,  and  ensures  fairness 
for  the  competing  offerors.  The  validation 
process  is  particularly  important  when  the 
negative  infojmation  is  provided  by  only 
one  reference,  or  when  there  is  any  doubt 


concerning  the  accuracy  of  the  informa¬ 
tion.  It  is  noted,  however,  that  while  the 
government  must  disclose  past  perfor¬ 
mance  problems  to  offerors’  it  need  not 
disclose  the  identity  of  its  sources. 

A  special  problem  arises  with  respect 
to  subcontractors.  Past  performance 
information  pertaining  to  a  subcontractor 
cannot  be  disclosed  to  a  private  party 
without  the  subcontractor’s  consent 
(OFPP  Policy  Letter  92-5,  Dec.  30,  1992). 
Because  a  prime  contractor  is  a  private 
party,  the  government  needs  to  obtain  the 
subcontractor's  consent  before  disclosing 
its  past  performance  information  to  the 
prime  during  negotiations.  There  are  a 
variety  of  ways  to  obtain  subcontractor 
consent.  For  example,  the  solicitation 
could  require  the  prime  to  submit  its 
subcontractor’s  consent  along  with  the 
prime’s  proposal  to  the  government. 


How  To  Treat  Past  Performance  Information 


PRAG  information  concerning  the  past 
performance  of  an  offeror  or  of  its  pro¬ 
posed  subcontractors  should  be  treated  as 
sensitive  source  selection  information. 
This  information  sometimes  includes 
information  that  is  proprietary,  such  as 
trade  secrets  and  confidential  commercial 
or  financial  data  that  would  not  be  re¬ 
leased  under  the  Freedom  of  Information 
Act.  Current  laws,  regulations,  and 
policies  governing  storage,  access,  disclo¬ 
sure,  and  marking  of  source  selection  and 
proprietary  information  must  be  observed 


at  all  times.  Questions  concerning  the 
procedures  for  the  handling  of  past  perfor¬ 
mance  information  should  be  referred  to 
the  contracting  officer  or  legal  counsel  for 
resolution. 

The  PRAG  must  retain  the  records  of 
its  activity  throughout  the  source  selection 
process.  Upon  contract  award  or  cancella¬ 
tion  of  the  solicitation,  all  PRAG  records 
are  provided  to  the  contracting  officer  for 
retention  along  with  the  other  source 
selection  documents. 


How  To  Improve  The  PRAG  Methodology 


The  PRAG  methodology  is  a  dynamic 
process  that  will  evolve  as  our  needs 
change  and  as  our  knowledge  base  ex¬ 
pands.  It  is  important  for  all  of  us  to 
share  information  and  ideas  to  ensure 
that  this  handbook  remains  current  and 
useful. 


Each  MSC  should  establish  a  central¬ 
ized  focal  point  to  capture  and  preserve  the 
lessons  learned  from  its  PRAGs.  Future 
AMC  workshops  will  call  upon  the  com¬ 
mands  to  share  their  experiences  and 
improve  this  handbook. 


“We  owe  it  to  our  soldiers  to  ensure  that  the  past  perfor¬ 
mance  of  our  contractors,  good  or  bad,  is  meaningfully 
considered  during  the  source  selection  process 

—  Commanding  General, 
U.S.  Army  Materiel 
Command  ; 


Appendix 


APPENDIX  A 

Sample  Section  M  Solicitation  Provision 


M.l  Evaluation  Factors  For  Award 

(a)  Selection  of  an  offeror  for  award  will  be  based  on  an  evaluation  of  proposals  in  three 
factors:  Technical,  Cost,  and  Performance  Risk.  Each  factor  is  separately  described  below  in 
greater  detail.  The  technical,  cost,  and  performance  risk  factors  will  not  be  numerically  scored 
but  rather  will  be  rated  in  an  adjectival  and  narrative  manner.  The  ultimate  objective  of  the 
evaluation  is  to  determine  which  proposal  offers  the  best  prospect  for  optimum  attainment  of 
the  objectives  of  this  program.  Negotiations  may  be  conducted  with  those  offerors  determined 
to  be  in  a  competitive  range  by  the  contracting  officer. 

(b)  The  technical  factor  is  slightly  more  important  than  the  cost  factor  which  is  slightly 
more  important  than  the  performance  risk  factor.  However,  to  be  considered  for  award  an 
offeror  must  be  determined  to  be  acceptable  in  the  technical  factor.  A  deficiency  could  consti¬ 
tute  a  basis  for  rejection  of  a  proposal.  Award  will  be  made  to  that  offeror  whose  proposal 
represents  the  best  overall  buy  for  the  government.  The  government  reserves  the  right  to 
award  to  other  than  the  low  offeror. 

(c)  Offerors  are  urged  to  ensure  that  their  proposals  are  submitted  on  the  most  favor¬ 
able  terms  in  order  to  reflect  their  best  possible  potential,  since  less  than  the  best  potential 
could  result  in  exclusion  of  the  proposal  from  further  consideration. 

Offerors  are  reminded  that  unsupported  promises  to  comply  with  the  contractual  re¬ 
quirements  will  not  be  sufficient.  Proposals  must  not  merely  parrot  back  the  contractual 
requirements  but  rather  must  provide  convincing  documentary  evidence  in  support  of  any 
conclusionary  statements  relating  to  promised  performance. 

(d)  The  offeror’s  proposal  is  presumed  to  represent  its  best  efforts  to  respond  to  the 
solicitation.  Any  inconsistency,  whether  real  or  apparent,  between  promised  performance  and 
price  should  be  explained  in  the  proposal.  Unexplained  inconsistencies  resulting  from  the 
offeror’s  lack  of  understanding  of  the  nature  and  scope  of  the  work  required  may  be  grounds  fur- 
rejection  of  the  proposal. 

M.2  Technical  Factor 

M.3  Cost  Factor 

M.4  Performance  Risk  Factor 

(a)  During  the  source  selection  process,  the  government  will  assess  the  relative  risks 
.associated  with  each  offeror  and  proposal.  It  is  important  to  note*  the  distinction  between 
proposal  risk  and  performance  risk. 

(1)  Proposal  risks  are  those  associated  with  an  offeror's  promised  approach  in  meeting 
the  government’s  requirements.  Proposal  risk  is  assessed  by  the  proposal  evaluators  and  is 
integrated  into  the  rating  of  each  specific  evaluation. subfactor  under  the  technical  and  cost 
factors. 
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(2)  Performance  risks  are  those  associated  with  an  offeror’s  likelihood  of  success  in 
performing  the  solicitation’s  requirements  as  indicated  by  that  offeror's  record  of  past  perfor¬ 
mance.  Performance  risk  is  assessed  by  the  PRAG  and  is  assigned  a  narrative  rating  in  the 
performance  risk  factor  of  the  evaluation. 

(b)  The  government  will  conduct  a  performance  risk  assessment  based  upon  the  qual¬ 
ity  of  the  offeror’s  past  performance  as  well  as  that  of  its  proposed  subcontractors,  as  it  relates 
to  the  probability  of  successful  accomplishment  of  the  required  effort.  When  assessing  perfor¬ 
mance  risk,  the  government  will  focus  its  inquiry  on  the  past  performance  of  the  offeror  and  its 
proposed  subcontractors  as  it  relates  to  all  solicitation  requirements,  such  as  cost,  schedule, 
and  performance,  including  the  contractor’s  record  of  conforming  to  specifications  and  to 
standards  of  good  workmanship;  the  contractor’s  record  of  containing  and  forecasting  costs  on 
any  previously  performed  cost  reimbursable  contracts;  the  contractor’s  adherence  to  contract 
schedules,  including  the  administrative  aspects  of  performance;  the  contractor’s -history  for 
reasonable  and  cooperative  behavior  and  commitment  to  customer  satisfaction;  and  generally, 
the  contractor’s  business-like  concern  for  the  interests  of  its  customers. 

(c)  A  significant  achievement,  problem,  or  lack  of  relevant  data  in  any  element  of  the 
work  can  become  an  important  consideration  in  the  source  selection  process.  A  negative  find¬ 
ing  under  any  element  may  result  in  an  overall  high  performance  risk  rating.  Therefore, 
offerors  are  reminded  to  include  all  relevant  past  efforts,  including  demonstrated  corrective 
actions,  in  their  proposal.  The  lack  of  a  performance  record  may  result  in  an  unknown  perfor¬ 
mance  risk  rating.  * 

(d)  Offerors  are  cautioned  that  in  conducting  the  performance  risk  assessment,  the 
government  may  use  data  provided  by  the  offeror  in  its  proposal  and  data  obtained  from  other 
sources.  Since  the  government  may  not  necessarily  interview  all  of  the  sources  provided  by  the 
offerors,  it  is  incumbent  upon  the  offeror  to  explain  the  relevance  of  the  data  provided. 

Offerors  are  reminded  that  while  the  government  may  elect  to  consider  data  obtained  from 
other  sources,  the  burden  of  providing  thorough  and  complete  past  performance  information 
rests  with  the  offerors. 

*'  Alternatively,  the  contracting  officer  may  elect  to  state  :  " The  lack  of  a  performance 
record  may  result  in  a  high  performance  risk  rating. "  (See  page  6,  " How  to  Evaluate 
No  Past  Performance.  ’’) 


“I  believe  that  the  consideration  of  past  performance  is  helping  the 
Army  make  better  source  selection  decisions.  The  risk  assessments 
add  another  dimension  to  our  ability  to  identify  the  hidden  risks  in 
an  otherwise  attractive  paper  proposal.  This  has  also  enhanced  the 
quality  of  our  negotiations.  ” 

—  Assistant  Secretary  of  the  Army 
(Research,  Development  and  Acquisition) 


APPENDIX  B 

Sample  Section  L  Solicitation  Provision 


( Caution .  Proposals  that  fail  to  contain  the  information  requested  in  this  paragraph  ma\ 
be  rejected  by  the  government.) 

Performance -  ihsk: 

The  offeror  shall  submit  a  description  of  its  previous  government  contracts  (all  prime  and 

major  subcontracts  received,  or  in  performance,  during  the  past  _ years)  which  are  in  any 

way  relevant  to  the  effort  required  by  this  solicitation.  Commercial  contracts  may  be  included 
if  necessary.  The  description  shall  include  the  following  information  in  the  following  format. 

•  Identify  in  specific  detail  for  each  previous  contract  listed,  why  or  how  you  consider  that 
effort  relevant  or  similar  to  the  effort  required  by  this  solicitation 

•  Your  (or  your  subcontractor’s)  CAGE  and  DUNNS  numbers 

•  Government  or  commercial  contracting  activity,  address,  and  telephone  number 

•  Procuring  Contracting  Officer’s  (PCO’s)  name  and  telephone  number 

•  Government  or  commercial  contracting  activity  technical  representative,  or  COR.  and 
telephone  number 

•  Government  or  commercial  contract  administration  activity,  and  the  name  and  tele¬ 
phone  number  of  the  Administrative  Contracting  Officer  (AGO) 

•  Contract,  number 

•  Contract  award  date 

•  Contract  type 

•  Awarded  price/cost 

•  Final,  or  projected  final,  price/cost 

•  Original  delivery  schedule 

•  Final  or  projected  final,  delivery  schedule 

•  A  narrative  explanation  on  each  previous  contract  listed  describing  the  objectives 
achieved  and  any  cost  growth  or  schedule  delays  encountered.  For  an>  government  contracts 
which  did  not/do  not  meet  original  requirements  with  regard  to  eiiher  cost,  schedule,  or  techni¬ 
cal  performance,  a  brief  explanation  of  the  reason(s)  for  such  shortcomings  and  any  demon¬ 
strated  corrective  actions  taken  to  avoid  recurrence.  The  offeror  shall  also  provide  a  copy  of 
any  cure  notices  or  show  cause  letters  received  on  each  previous  contract  listed  and  a  descrip¬ 
tion  of  any  corrective  action  by  the  offeror  or  proposed  subcontractor. 

•  The  offeror  shall  also  provide  the  above  required  information  for  any  and  all  contracts  it 

has  had  terminated  in  whole  or  in  part,  for  default  during  the  past _ years,  to  include  .hose 

currently  in  the  process  of  such  termination  as  well  as  those  which  are  not  similar  to  tlu- 
.proposed  effort.  The  contractor  shall  list  each  time  the  delivery  schedule  was  revised  and 
provide  an  explanation  of  why  the  revision  was  necessary. 
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•  New  corporate  entities  may  submit  data  on  prior  contracts  involving  its  officers  and 
employees.  However,  in  addition  to  the  other  requirements  in  this  section,  the  offeror  shall 
discuss  in  detail  the  role  performed  by  such  persons  in  the  prior  contracts  cited. 

•  Offerors  shall  provide,  an  outline  of  how  the  effort  required  by  the  solicitation  will  be 
assigned  for  performance  within  the  contractor's  corporate  entity  and  among  proposed  subcon¬ 
tractors.  Information  required  in  the  above  paragraphs  shall  be  provided  for  each  proposed 
subcontractor  who  will  perform  a  significant  portion  of  the  effort.  “Significant”  is  defined  for 
these  purposes  in  terms  of  estimated  dollar  amount  of  the  subcontract  (e.g.,  $1,000,000  or 
more)  and/or  in  terms  of  criticality  of  the  subcontracted  work  to  the  whole.  With  regard  to 
prime  contract  assignments  that  will  be  performed  by  you  and  not  a  proposed  subcontractor, 
you  shall  indicate: 

1)  wliaUnternal  corporate  bodies/divisions  will  accomplish  which  portions  of  the  effort, 

2)  whether  or  not  those  divisions  were  responsible  for  performance  under  the  previous  con¬ 
tracts  cited  for  the  instant  proposal,  and 

3)  if  those  divisions  have  relocated  since  the  accomplishment  of  previous  cited  contract  efforts, 
a  description  of  any  changes  arising  from  that  relocation  in  terms  of  key  personnel,  facilities 
and  equipment. 

•  Offerors  shall  include  in  their  proposal  the  written  consent  of  their  proposed  significant 
subcontractors  to  allow  the  government  to  discuss  the  subcontractor's  past  performance  evalu¬ 
ation  with  the  offeror  during  negotiations. 

Note:  Offerors  are  reminded  that  both  independent  data  and  data  provided  by  offerors  in 
their  proposals  may  be  used  to  evaluate  offeror  past  performance.  Since  the  government  may 
not  necessarily  interview  all  of  the  sources  provided  by  the  offerors,  it  is  incumbent  upon  the 
offeror  to  explain  the  relevance  of  the  data  provided.  The  government  does  not  assume  the 
duty  to  search  for  data  to  cure  problems  it  finds  in  proposals.  The  burden  of  providing  thor¬ 
ough  and  complete  past  performance  information  remains  with  the  offerors.  Proposals  that  do 
not  contain  the  infoi matiun  lequested  by  tbit  pai  agi  aph  ribk  rejection  or  high  risk  rating  by 
the  government. 


“The  purpose  of  PRAG  is  to  evaluate  performance  risk  — 
to  seek  out  and  reward  good  performance.  ” 

—  Deputy  Commanding  General 
U.S.  Army  Materiel  Command 


APPENDIX  C 

Sample  PRAG  Assessment  Plan 


Definitions 

Performance  Risks: 

Performance  risks  are  those  associated  with  an  offeror's  likelihood  ofsuccos.s  in  perform¬ 
ing  the  solicitation's  requirements  as  indicated  by  that  offeror’s  record  of  past  |x?rformance 
Performance  risk  is  assessed  by  the  Performance  Risk  Assessment  Group  (PRAG)  and  is 
assigned  a  narrative  rating  in  the  Performance  Risk  Factor  of  the  evaluation. 

Performance  Risk  Assessment  Group: 

A  PRAG  is  the  source  selection  entity  that  assesses  performance  risk.  The  PRAG  may 
either  be  separate  from  the  SSEB  and  report  directly  to  the  SSAC,  or  operate  as  a  separate 
group  within  the  SSEB  and  report  through  the  SSEB  chairperson  to  the  SSAC.  Each  contract¬ 
ing  activity  determines  the  appropriate  composition  and  structure  of  its  PR\Gs.  depending 
upon  the  size,  nature,  and  complexity  of  a  particular  procurement. 

Proposal  Risks: 

Proposal  risks  are  those  associated  with  an  offeror's  proposed  approach  in  meeting  the 
government’s  requirements.  Proposal  risk  is  assessed  by  the  proposal  evaluators  and  is  inte¬ 
grated  into  the  rating  of  each  specific  evaluation  subfactor  under  the  technical  and  cost  factors 

The  Performance  Risk  Assessment  Group 

Responsibiliti  es : 

The  PRAG  shall  perfoim  an  in-depth  review  and  evaluation  of  the  performance  data 
provided  by  offerors  and  obtained  from  other  sources  to: 

•  Assess  each  offeror’s  past  and  current  performance  as  it  relates  to  the  solicitation 
requirements.  The  PRAG  should  consider  the  relevancy,  recency  and  accuracy  of  the  data  in 
arriving  at  its  overall  assessment. 

•  Identify  strong  and  weak  points  for  use  during  negotiations  and/or  contract  administra¬ 
tion. 


Performance  Risk  Assessment: 

The  performance  risk  assessment  conducted  by  the  PRAG  assesses  each  offeror’s  record  of 
performance  to  determine  the  offeror's  likelihood  of  success  in  performing  the  required  effort 
The  PRAG  must  focus  its  inquiry  on  the  offeror's  record  of  performance  as  it  relates  to  the 
performance  of  the  solicitation  requirements.  Therefore,  the  PRAG  must  become  thoroughly 
familiar  with  the  statement  of  work  and  specifications.  Since  the  PRAG  does  not  jx'rform  the 
proposal  risk  assessment  (the  SSEB’s  proposal  evaluators  do  that),  it  does  not  normally  review 
the  offerors  proposals. 

The  PRAG's  performance  risk  assessment  is  not  solely  limited  to  the  prime  contractor 
division  submitting  the  proposal  v*'hen  other  divisions,  corporate  entities,  critical  subcontrac¬ 
tors,  or  teaming  contractors  perform  a  critical  element  of  the  required  effort.  In  such  cases,  the 
PRAG  should  evaluate  the  other  organization’s  performance  record. 
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Each  performance  risk  assessment  will  consider  the  number  and  severity  of  problems,  the 
effectiveness  of  corrective  actions  taken,  and  the  overall  work  record.  The  assessment  of 
performance  risk  is  not  intended  to  be  the  product  of  a  mechanical  or  mathematical  analysis  of 
an  offeror's  performance  on  a  list  of  contracts,  but  rather  the  product  of  subjective  judgment  of 
the  PRAG  after  it  considers  all  available,  relevant  and  recent  information.  The  following 
definitions  of  performance  risk  should  be  used: 

•  High  Performance  Risk  *  Based  on  the  offeror's  performance  record,  significant  doubt 
exists  that  the  offeror  will  successfully  perform  the  required  effort. 

•  Moderate  Performance  Risk  -  Based  on  the  offeror’s  performance  record,  some  doubt 
exists  that  the  offeror  will  successfully  perform  the  required  effort. 

•  Low  Performance  Risk  -  Based  on  the  offeror's  performance  record,  little  doubt  exists 
that  the  offeror  will  successfully  perform  the  required  effort. 

•  Unknown  Performance  Risk  -  No  performance  record  identifiable  (this  category  is 
optional). 

Note:  Each  of  the  high,  moderate,  and  low  risk  categories  may  be  further  subdivided  to 
enhance  the  comparative  analysis  of  offerors. 

I Insert  Section  M  of  the  Solicitation  here] 

Documentation: 

•  The  PRAG’s  performance  •  k  assessment  report  will  be  provided  directly  to  the  con¬ 
tracting  officer.  The  results  of  tli  j  PRAG  evaluation  will  also  be  briefed  to  the  contracting 
officer  by  the  PRAG  chairperson. 

•  The  PRAG’s  documentation  and  presentations  should  address  the  following: 

—  The  sources  of  the  performance  data 

—  The  relevancy  of  the  data  to  die  program  requirements 

—  The  currency  of  the  data 

—  The  performance  risk  assessment  of  each  offeror 

—  The  supporting  rationale  for  each  performance  risk  assessment 

—  The  strong  and  weak  areas  of  each  offeror  for  use  during  negotiations  and/or  con¬ 
tract  administration 

—  Any  other  matters  deemed  relevant 

Gathering  Performance  Data 

•  The  two  primary  sources  of  performance  data  are  the  contractor  references  contained  in 
the  performance  risk  volume  of  the  offeror's  proposal  and  the  AMC  CIS. 
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•  The  CIS  does  not  contain  a  subjective  analysis  of  past  performance.  Instead,  it  was 
designed  to  provide  the  PRAG  with  the  key  factual  information  needed  to  commence  an  in\es- 
tigation  into  the  contractor's  performance  history.  The  actual  assessment  has  been  i  Lsencd 
for  the  PRAG  members  who  can  best  determine  which  information  is  most  rele\ant  to  tin- 
acquisition. 

•  Upon  request,  the  CIS  will  provide  the  PRAG  with  a  Contractor  Information  Report 
(CIR)  on  an  offeror's  contract  history  within  AMC.  In  reviewing  that  data  report,  the  PRAG 
should  exercise  its  own  judgment  to  determine  which  of  ti.c  offeror's  past  efforts  are  most 
relevant  to  the  solicitation's  requirements.  Key  points  of  contact  will  be  identified  on  the  data 
report  for  direct  telephonic  contact. 

•  The  PRAG  will  obtain  whatever  information  it  deems  most  relevant  to  the  required 
effort  by  telephonic  and/or  written  inquiry  with  the  points  of  contact  identified  on  the  CIR.  It 
is  important  that  each  discussion  be  accurately  summarized  on  a  PRAG  Telephone  lnten  iew 
Report  Form  for  it  is  this  material  which  will  later  serve  as  back-up  for  the  PRAG's  perfor¬ 
mance  risk  assessment.  A  copy  of  the  Telephone  Jntcnicw  Report  Form  must  he  promptly,  sent 
by  certified  mail  (return  receipt  requested )  or  by  data  fax  to  the  point  of  contact  fur  i  erifh  utiuii. 
There  is  usually  no  need  to  divulge  the  solicitation  number,  or  other  identifying  information  to 
the  reference.  If  you  do  so,  however,  you  need  to  obtain  a  nondisclosure  statement. 

•  The  PRAG  should  also  exercise  its  judgment  in  determining  which,  if  am .  of  the  con¬ 
tractor  supplied  references  should  be  called  fur  additional  information  or  \  erification.  Roth 
negative  and  positive  information  should  be  corroborated  before  it  is  relied  upon  to  ain  .signifi¬ 
cant  degree  to  ensure  accuracy  in  the  final  PRAG  report  and  fairness  in  the  overall  process. 
PRAG  Telephone  Interview  Report  Forms  should  be  completed  for  these  contacts  as  well. 

•  The  key  to  the  success  of  each  performance  risk  assessment  is  the  PRAG's  w  illingness 
and  ability  to  seek  out  the  most  relevant,  recent,  and  accurate  information  available.  Should  a 
PRAG  member  be  unable  to  obtain  information  for  a  reference,  he  or  she  may  contact  the 
PRAG  chairperson  who  should  seek  assistance  through  the  source  selection  hierarch; . 


“I  was  the  contracting  oJJicer...PRAC  ivas  a  new  initiative 
at  that  time.  I  used  it  and  it  worked!  The  contractor  deliv¬ 
ered  a  quality  product,  on  time  and  within  budget.  The 
users  were  thrilled  with  it  and  it  also  played  a  major  role 
in  Desert  Storm.” 

—  Contracting  Officer,  U.S.  Army  Communications- 
Electronics  Command,  Vint  Hill  Farms  Station 
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Sample  Format  For  Telephone  Intermew  Report 

PRAG  I.D.  NUMBER: 

CONTRACTOR:  (Name  &  Address) 

PERSON  CONTACTED:  (Name,  Address,  Phone  #) 

DATE  &  TIME  OF  CONTACT: 

lam  (name).  My  telephone  number  is  (#).  I  am  calling  in  reference  to  contrac¬ 
tor  (name).  My  questions  will  pertain  to  that  contractor's  record  of  past  and  current  perfor¬ 
mance.  The  information  that  you  provide  will  be  used  in  the  awarding  of  federal  contracts. 
Therefore  it  is  important  that  your  information  be  as  factual  and  accurate  as  possible.  A 
summary  of  this  discussion  will  be  sent  to  you  for  your  records.  If  that  summary  is  inaccurate 
or  incomplete  in  any  way,  please  contact  me  immediately. 

SUMMARY  OF  DISCUSSION: 


Signature  of  PRAG  Member 


Telephone  # 


Reminders  for  PRAG  Member: 

•  Discuss  recency  and  relevance  of  information 

•  Read  summary  to  person  contacted 

•  Send  copy  to  person  contacted 

•  Withhold  the  identity  of  your  program  and  solicitation  number,  if  practicable,  to  avoid 
having  to  obtain  a  non-disclosure  statement  from  the  person  contacted 
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Sample  Format  For  Telephone  Interview  Report  Cover  Letter 


I  On.  letterhead J 


Address 


1.  Attached  is  a  summary  of  your  telephone  conversation  with  a  member  of  the  Perfor¬ 
mance  Risk  Assessment  Group  on  (date). 

2.  If  this  office  does  not  hear  from  you  by  ( dale >  we  will  assume  that  the  summary  of  the 
discussion  is  correct.  If  you  have  any  questions  or  comments  you  may  address  them  to 
(address).  You  may  also  call  me  directly  at  (phone  $). 

3.  We  thank  you  for  your  time  and  assistance  regarding  this  effort. 


Enel. 


Chairman.  PRAG 


“We  need  to  remind  our  contractors  that  today’s  performance 
is  tomorrow’s  past  performance,  and  we  will  hold  them 
accountable  for  their  past  performance  in  our  source  selec¬ 
tions.  ” 

—  Command  Counsel, 

U.S.  Army  Materiel  Command 


APPENDIX  F 

Sample  Questions  And  Ideas  For  Telephone  Interviews  And 

Questionnaires 


•  Because  the  word  “PRAG”  is  new,  it  may  not  be  recognized  by  the  references.  To  avoid 
confusion',  simply  state  that  thevontractor's  past  performance  is  being  assessed  for  a  source 
selection.  There  is  usually  no  need  to  divulge  the  solicitation  number,  program  description,  or 
other  identifying  information  to  the  reference.  If  you  do  so,  you  need  to  obtain  a  nondisclosure 
statement. 

•  Confirm  the  following  data  received  from  the  contractor: 

—  Contract  number  and  type 

—  Award  amount  and  final  or  projected  fins!  amount 
—  Award  delivery  schedule  and  final  or  projected  final  delivery  schedule 
—  Nature  of  the'bffort  (i.e.,  the  scope  of  the  effort,  the  types  of  tasks  involved  and  the 
product  to  be  delivered) 

•  Iffthe  award  amount  or  delivery  schedule  has  changed,  find  out  what  caused  the  change. 

•  Discover  what  role  the  reference  played  (e.g.,  COR,  contract  specialist,  ACO,  etc.)  and 
for  how  long. 

•  If  a  problem  is  uncovered,  discuss  what  the  government  and  contractor  did  to  resolve  it. 

•  Ask  for  a  description  of  the  types  pf  personnel  (skills  and  expertise)  the  contractor  used 
and  the  overalkquality  of  the  contractor's  team.  Did  the  company  appear  to  use  personnel 
witlnthe  appropriate  skills  and  expertise? 

•  Ask  how  the  contractor  performed  consideri  ng  quality  of  performance,  responsiveness, 
schedule,  overall  management,  technical  performance,  and  financial/cost  management. 

-•  Ask  whether  the  contractor  was  cooperative  in  negotiations  and  in  resolving  issues. 

•  Inquire  whether  there  were  any  particularly  significant  risks  involved  in  performance  of 
the  effort. 

•  Ask  if  the  company  appeared  to  apply  sufficient  resources  (personnel  and  facilities)  to 
the  effort. 

•  Ask  if  the  company  used  subcontractors.  If  so,  what  was  the  relationship  between  the 
prime  and  the  subcontractors?  What  was  the  management  role  of  the  prime  and  how  well  did 
it  manage  the  subcontractors?  Did  the  subcontractors  perform  the  bulk  of  the  effort  or  just  add 
breadth  or  depth  on  particular  technical  areas?  If  the  subcontractors  worked  on  specific  tech¬ 
nical-areas,  what  were  those  areas  and  why  were  they  accomplished  by  the  subcontractors 
rather  than  the  prime? 

•  If  a  problem  is  uncovered  that  the  reference  is  unfamiliar  with,  ask  for  another  indi¬ 
vidual  who  might  have  the  information. 

•  Inquire  whether  there  are  other  past  efforts  by  this  firm  with  the  reference's  agency. 

•  Inquire  what  the  company's  strong  points  are  or  what  the  reference  liked  the  most 
about  then.. 

•  Inquire  what  the  company's  weak  points  are  or  what  the  reference  liked  least  about 
them. 

•  Inquire  whether  the  reference  has  any  reservations  about  recommending  a  future 
contract  award  to  this  company. 

•  Inquire  whether  the  reference  knows  of  anyone  else  who  might  have  past  performance 
information  on  the  offeror. 
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Sample  Questionnaire 


Program  Name: 

Contract  No.: 

1.  Please  specify  the  contract  requirements,  purpose,  and  technology. 


2.  Did  the  contractor  meet  the  original  equipment  performance  requirements?  Please 
explain: 


3.  Did  the  contractor  request  specification  relief?  If  so.  was  there  an  impact  on  system 
performance,  cost  or  delivery? 


4.  Did  the  contractor  use  Ada  language?  If  yes.  did  the  contractor  meet  the  Ada  language 
and  software  requirements?  Please  explain: 

5.  Did  the  contractor  meet  test  schedule  requirements?  Please  explain: 

6.  Were  any  Quality  Deficiency  Reports  (QDR)  or  corrective  action  requests  submitted  to 
the  contractor  due  to  quality  deficiencies?  Please  explain: 


7.  Opinion:  Quality,  reliability  and  maintainability  of  equipment  delivered.  Very  Good  1  1 
Good  ( )  Acceptable  ( )  Marginally  Acceptable  ( )  Please  explain: 

8.  Was  the  contractor's  engineering  management  effective  in  controlling  costs,  schedule 
and  performance  requirements?  Please  explain: 

9.  Did  the  contractor  successfully  manage  its  subcontractors?  Please  explain. 

10.  Was  human  engineering/manprint  a  requirement?  If  so.  was  it  satisfactory?  Please 
explain: 

11.  Was  logistics  support  satisfactory  in  meeting  contract  requirements?'  Please  explain : 
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12.  At  completion  of  the  contract,  was  the  contractor  committed  to  customer  satisfaction? 
Please  explain: 

13.  Rate  the  contractor  s  overall  technical  performance:  Good  (  )  Fair  ( )  Poor  ( )  Would 
you  recommend  this  contractor  for  other  contracts?  Please  explain: 


14.  Were  there  any  problems  with  Engineering  Change  Proposal,  Requests  for  Waivers,  or 
Requests  for  Deviations?  Please  explain: 


15.  Were  there  any  problems  with  Logistics  Support  Documentation?  Please  explain: 


16.  During  technical  meetings,  was  the  contractor  cooperative  and  receptive  to  govern¬ 
ment  concerns  affecting  production  and/or  performance  requirements?  Please  explain: 


17.  With  respect  to  design,  engineering  capability,  and  overall  technical  performance, 
would  you  recommend  this  contractor  for  similar  government  contracts?  Please  explain: 


18.  How  would  you  rate  the  contractor's  technical  performance  on  this  contract? 
Outstanding  ( )  Good  ( )  Poor  ( )  Please  explain: 


19:  Do  you  know  of  anyone  else  who  might  have  relevant  information  concerning  this 
’contractor’s  past  performance?  Please  explain: 


20.  Please  make  any  additional  comments  you  wish  here: 
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